@Bultako - Thanks!

I think for now, it might be easier to script the copy of the files from $GAMMAPY_EXTRA to the docsfolder (if really needed, ifnbsphinxcan't be configured to directly read the notebooks from there) indocs/conf.py, or really in a function in gammapy/utils/docs.pythat you then call fromdocs/conf.py` like here:

Of course, yes, long term I would like to have an option for python setup.py build_docs that allows executing different steps of the docs build separately: especially to have the notebook part separately. But that's not a requirement for this to get merged. If you want to make it configurable now, you could use an environment variable GAMMAPY_EXECUTE_NBSPHINX and access that via os.environ. It would be nice to keep that part optional, if it's possible to script that way.

Is there some way to run the build to get "verbose" or "debug" output about what sphinx or nbsphinx is doing? I guess at the moment it's not possible to just execute the new nbsphinx part of the build, to debug and understand it?

My suggestion would be that you do this first, to avoid having to have the notebooks in the gammapy repo like you have now. The thing is that having all the large files makes the PR hard to review (diff so large) and means I can't merge (no large files allowed in the Gammapy repo).

The other thing that's needed to make this PR "mergeable" is to squash the merge commits and commits that contain the large notebook files:
https://github.com/gammapy/gammapy/pull/1176/commits

If you're not sure how to do this with git, let me know. This explanation is pretty good:
http://astropy.readthedocs.io/en/stable/development/workflow/development_workflow.html#how-to-squash

One more thing on your TODO list:

Rename index.ipynb -> notebooks.ipynb

I think really it would be better to change index.ipynb to an RST page? It doesn't contain any code, and RST is better for an index / overview page than a notebook, no?
Not sure if this needs to be done in this first PR or if this can be done later.

I'll have a closer look at this tomorrow.

I've fixed a few more notebook formatting issues in gammapy-extra, and I've fixed the Sphinx docs build in master by adding this missing dependency: 71dd5c2

@Bultako - Could you please add nbsphinx to .travis.yml so that we start to see it running in CI on this pull request (this should fail: https://travis-ci.org/gammapy/gammapy/jobs/290333660), as well as to docs/environment.yml so that it'll work on RTD when we merge the PR?

Locally I see the following warnings from the Sphinx build:
https://gist.github.com/cdeil/23072fba2afeb652b2427bda787e06ee
We need to get rid for all of them - the Sphinx build on travis-ci is configured such that it fails if there are warnings.

Do you know if there is a way to use the line number information in these warnings somehow to find the problematic notebook cell / line that needs to be fixed?

/Users/deil/code/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.

Is the plan that notebooks will be copied to docs/notebooks instead of directly to the docs/_build output folder? If yes, then please add the docs/notebooks folder to the make clean command here:

My last comment with the warnings at https://gist.github.com/cdeil/23072fba2afeb652b2427bda787e06ee was incorrect: I think the warnings like

/Users/deil/code/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.

should be fixed now. I just didn't realise I was running on an old version of the notebooks because make clean and build_docs doesn't do the clean and copy of notebooks from GAMMAPY_EXTRA yet.

@cdeil

• YAML files and Makefile fixed

I think the plan is still copying notebooks from $GAMMAPY_EXTRA to docs/notebooks.
There are still some warnings related with orphan notebooks (not present in any doctree directive) that may be solved with a cleaned notebooks.rst file, I will check it and commit later.

Locally I still get these warnings:

Users/deil/code/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/Users/deil/code/gammapy/docs/notebooks/cta_1dc_introduction.ipynb:1657: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/Users/deil/code/gammapy/docs/notebooks/cta_data_analysis.ipynb:9: WARNING: Explicit markup ends without a blank line; unexpected unindent.
WARNING: /Users/deil/code/gammapy/docs/notebooks/data_fermi_lat.ipynb:: (ERROR/3) Anonymous hyperlink mismatch: 1 references but 0 targets.
See "backrefs" attribute for IDs.

The CI build hasn't started ... the main goal for this PR would be to get green light there:
https://travis-ci.org/gammapy/gammapy/jobs/290437970

@Bultako - Do you have time to work on this PR in the coming days or is this ready to be merged from your side? I would prefer to merge this soon to see if it works on RTD, and to move the task list from above to a follow-up issue / pull request.

@cdeil

I think we were working in different versions of gammapy-extra

• I have pulled the last master version gammapy-extra 8359164
• I have done a make clean in gammapy root folder
• I have done a python setup.py build_docs and got a warning for data_fermi_lat.ipynb

There is a broken link in that nb, I have fixed and rebuild the docs, this time I had no warnings..
I have made a PR in gammapy-extra to fix this.
gammapy/gammapy-extra#89

Re: Travis CI green light.
It blocked because of IPython needed for syntax highlighting in notebooks. I have edited the YAML files. Crossing fingers...

We can merge this PR as it is when we'll have green light, next week I'll be a bit busy.

Merging now. I want to do some more changes on this, but let's see if this is working on RTD at all.

It worked! 🎉
@Bultako - Thank you!

• http://docs.gammapy.org/en/latest/notebooks.html

The build log is at http://readthedocs.org/projects/gammapy/builds/6156543/, and the timeout is a weird bug in the RTD docker infrastructure - all of our builds end in a timeout, even if it was successful. So for now one has to look through the log and see if there's any error, which in this case there isn't.

good :)
I see in http://docs.gammapy.org the search does not find any notebook, but locally in my html built docs the search works pretty well. do you have an idea on how to solve this? should I open an issue an have a look?

I tried this and it finds an entry:
http://docs.gammapy.org/en/latest/search.html?q=WMAP&check_keywords=yes&area=default
but it doesn't display correctly in the search results.

Do you have an example search URL you're using to test?

Let me try and wipe RTD to trigger a clean build now and see if that makes it better.

I've wiped RTD and triggered a new build here: https://readthedocs.org/projects/gammapy/builds/6162287/

@Bultako - Do you have an RTD account? If yes, I would be happy to give you admin permissions there for Gammapy.

@cdeil
http://docs.gammapy.org/en/latest/search.html?q=data+challenge
does work nicely locally but does not find the notebooks in RTD

I've just created my bultako account in RTD :)

@Bultako - You're now an admin on RTD for Gammapy.

We've had problems before with Sphinx search not showing certain things: #611

Maybe you could open a new issue in the Gammapy tracker? This one is closed and I don't have time to investigate today.

@cdeil

After some investigation I've found RTD always build an HTML version together with a JSON version of the docs. These are used for web serving & search indexing, respectively.
http://docs.readthedocs.io/en/latest/yaml-config.html?highlight=JSON

But, our last builds in RTD have not finished properly, so I suspect JSON format has not been built properly, as it is also the case with htmlzip, epub and pdf formats. Note that the latest version of the RTD documentation does not link properly to these formats
e.g. https://media.readthedocs.org/pdf/gammapy/latest/gammapy.pdf

You said you could have look at the build logs in RTD?
I have not found how to do this..